Precision luxmeter methods for digital cameras to quantify colors in uncontrolled lighting environments

ABSTRACT

In one embodiment, a diagnostic system for biological samples is disclosed. The diagnostic system includes a diagnostic instrument, and a portable electronic device. The diagnostic instrument has a reference color bar and a plurality of chemical test pads to receive a biological sample. The portable electronic device includes a digital camera to capture a digital image of the diagnostic instrument in uncontrolled lightning environments, a sensor to capture illuminance of a surface of the diagnostic instrument, a processor coupled to the digital camera and sensor to receive the digital image and the illuminance, and a storage device coupled to the processor. The storage device stores instructions for execution by the processor to process the digital image and the illuminance, to normalize colors of the plurality of chemical test pads and determine diagnostic test results in response to quantification of color changes in the chemical test pads.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of United States (U.S.) provisionalpatent application No. 62/038,155 entitled PRECISION LUXMETER METHODSWITH A DIGITAL CAMERA TO QUANTIFY COLORS IN UNCONTROLLED LIGHTINGENVIRONMENTS filed on Aug. 15, 2014 by inventor Bernard Burg.

This application is related to U.S. patent application Ser. No.14/633,518 entitled QUANTIFYING COLOR CHANGES OF CHEMICAL TEST PADSINDUCED BY SPECIFIC CONCENTRATIONS OF BIOLOGICAL ANALYTES UNDERDIFFERENT LIGHTING CONDITIONS filed on Feb. 27, 2015 by Bernard Burg etal. which is incorporated herein by reference for all purposes. U.S.patent application Ser. No. 14/633,518 claims priority to provisionalpatent application No. 61/948,536 entitled APPARATUS FOR DETERMININGANALYTE CONCENTRATION BY QUANTIFYING AND INTERPRETING COLOR INFORMATIONCAPTURED IN A CONTINUOUS OR PERIODIC MANNER filed on Mar. 5, 2014 byBernard Burg et al. (hereinafter Burg '536), which is incorporatedherein by reference for all purposes.

This application is also related to International Patent Ap. No.PCT/US2013/035397, Publication No. WO 2014025415 A2, filed on Aug. 5,2013 by Bernard Burg et al. (hereinafter Burg '397), and U.S. patentapplication Ser. No. 14/419,939 entitled METHOD AND APPARATUS FORPERFORMING AND QUANTIFYING COLOR CHANGES INDUCED BY SPECIFICCONCENTRATIONS OF BIOLOGICAL ANALYTES IN AN AUTOMATICALLY CALIBRATEDENVIRONMENT filed Feb. 6, 2015, both of which are incorporated herein byreference for all purposes. PCT Application No. PCT/US2013/035397 claimsthe benefit of U.S. provisional patent application No. 61/680,842entitled MULTI-ANALYTE RAPID DIAGNOSTIC TEST AND METHOD OF USE filed onAug. 8, 2012, by inventors Bernard Burg et al which is also incorporatedherein by reference for all purposes.

FIELD

The invention relates generally to systems and methods for measuringphotometric units in uncontrolled lighting environments with digitalcameras.

BACKGROUND

Illuminance E_(v) is the area density of luminous flux received by anilluminated body, integrated with all wavelengths and all directions.Illuminance is used to gauge the amount of light incident on a surface.A unit of illuminance is a lux (lx), lumens per square meter. Anotherunit of illuminance is foot candles. Illuminance on a surface area isluminous flux per area defined by the equation:

$E_{v} = {\frac{{luminous}\mspace{14mu}{flux}}{area} = \frac{{d\Phi}_{v}}{dA}}$

The spectral illuminance E_(v)(λ) is defined by the illuminance per unitwavelength interval at the wavelength λ. Spectral illuminance E_(v)(λ)is related to the illuminance E_(v) by the equationE _(v)=∫₀ ^(∞) E _(v)(λ)dλ.

A device or application that measures illuminance is referred to as aluxmeter. Luxmeter software applications that measure illuminance areavailable for use with portable electronic devices that include adigital camera. The luxmeter software application is executed by aprocessor and the operating system software (e.g., APPLE iOS and GOOGLEAndroid) of the portable electronic device. The accuracy of the luxmetersoftware applications is often limited by the sensor and digital cameraused in the portable electronic device. In addition, OS manufacturersmay only provide incomplete application programming interfaces (API) tothese sensors, or prohibit direct access by other applications to thesesensors. Consequently, luxmeter applications typically have limitedaccess to metadata of the digital pictures taken with a digital camera,if available. Typical metadata parameters produced by portableelectronic device cameras that are associated with digital photographsare:

-   -   ExposureTime;    -   ShutterSpeedValue;    -   FNumber;    -   ApertureValue;    -   BrightnessValue;    -   ISOSpeedRatings;

These metadata parameters are sometimes stored in registers as registersettings. The metadata parameters ExposureTime, ShutterSpeedValue,FNumber, and ApertureValue are well defined because they follow standardphotographic values. The parameter ExposureTime is related to theparameter ShutterSpeedValue through the mathematical equation:

${ExposureTime} = \frac{1}{2^{ShutterSpeedValue}}$The parameter FNumber is related to ApertureValue through themathematical equation:FNumber=√{square root over (2^(ApertureValue))}

The metadata parameter values of BrightnessValue and ISOSpeedRatingsvary from camera to camera, such that they are difficult to use for aluxmeter application. As a consequence, luxmeters in applications arequite approximate or require an external calibration before use.

A number of references disclosing measurement of illuminance conditionswere developed during the argentic film era, such as U.S. Pat. No.3,972,626, and U.S. Pat. No. 3,973,266, when film based cameras wereused to capture photos. Digital cameras revolutionized the structure ofcameras, capturing pixels of images and storing them into flash memory.A number of analog optical mechanisms were replaced with digitalcomputations. Accordingly, some inventions disclosed in patents, such asU.S. Pat. No. 5,737,648, were no longer relevant. New illuminancemeasurement methods were introduced for digital cameras.

U.S. Pat. No. 7,071,456 discloses a camera illuminance sensor to setlighting levels of I/O systems, such as keypads, and backlights foradjustable displays. U.S. Pat. App. Pub. No. 2012/0236173 similarlydiscloses adapting a camera user interface to environmental conditions.U.S. Pat. App. Pub. No. 2012/0236173 discloses use of a plurality ofsensors to allow corrections for underwater conditions such as very coldconditions, very bright conditions, and very dark conditions. U.S. Pat.App. Pub. No. 2012/0236173 discloses addressing illuminance conditionsof extreme brightness and extreme darkness, but does not make preciseilluminance measurements.

U.S. Pat. No. 7,629,998 and U.S. Pat. App. Pub. No. 2007/0002143disclose a method and apparatus for measuring the performance of acamera, including illuminance. A test environment is disclosed to bettercomprehend and verify the specifications of a camera and test theirfitness. The disclosed approach to measure illuminance requires thatadditional sensors be used for the test environment.

U.S. Pat. App. Pub. No. 2001/0007470 entitled MEASUREMENT OF ILLUMINANCECONDITIONS uses external light emitting diodes (LEDs), as well as colorfilters and photo sensors to measure illuminance. The LEDs are used tocontrol the lighting conditions. The intensity of each main color in theilluminance light is measured by dedicated photosensors havingcorresponding color measurement bands. These color intensities are usedto adjust signals originating from the charge coupled device (CCD) arrayof the photo image sensor. Using external light emitting diodes (LEDs)and color filters makes it more difficult to measure illuminance.

It is desirable to ease the measurement of illuminance so that it can bereadily used.

SUMMARY OF THE INVENTION

An illuminance model and additional measurements are introduced toincrease the accuracy of illuminance measurements and to further improvecolor quantification in digital photographs captured with portableelectronic device digital cameras, so they can be used with a medicaldevice under different lighting environments with unknown illuminance.

BRIEF DESCRIPTION OF THE DRAWINGS

This patent or application contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the United States Patent andTrademark Office upon request and payment of the necessary fee.

FIG. 1A illustrates a urine analysis test strip and an associatedmanufacturer interpretation color chart (MICC);

FIG. 1B illustrates a urine analysis paddle;

FIG. 2A is a block diagram of a system for analyzing a biological sampleof an analysis paddle, according to an embodiment;

FIG. 2B is an exemplary electronic device of the system for analyzingbiological samples.

FIG. 3A is a flow chart of a method of color quantification undercontrolled illuminance.

FIG. 3B is a graph illustrating color quantification using a singlecolor trajectory under controlled lighting conditions.

FIG. 4 is a graph of color trajectories for a plurality of differentlighting conditions and illuminances.

FIG. 5 is a flow chart of a method of color quantification under unknownilluminances;

FIG. 6 is an illustration of a diagnostic instrument with a referencecolor bar (RCB) for illuminance measurements;

FIG. 7 is a flow chart of a method of color quantification including theprocessing of the image of the reference color bar chart (RCB);

FIG. 8 illustrates a flow chart of a learning algorithm during alearning phase to acquire a mathematical model of the illuminanceswitch;

FIG. 9 illustrates a functional block diagram corresponding to thelearning phase and the determination of the mathematical model of theilluminance switch by the learning algorithm of FIG. 8;

FIG. 10 illustrates a functional block diagram corresponding to thetest/QA or measurement phase and the color quantification algorithm ofFIG. 7;

FIG. 11 illustrates a block diagram of the color quantification modelduring the learning phase; and

FIG. 12 illustrates the block diagram of the color quantification modelduring the test or measurement phase.

FIG. 13 illustrates a block diagram of a server for data mining aplurality of digital photos stored in a database on a data storagedevice.

FIG. 14 illustrates a number of objects that may be used to increase theaccuracy of the methods of lux measurements with a digital camera.

FIG. 15 illustrates an example of feature extraction using a driverlicense.

DETAILED DESCRIPTION

In the following detailed description, numerous specific details are setforth in order to provide a thorough understanding. However, it will beobvious to one skilled in the art that the embodiments may be practicedwithout these specific details. In other instances well known methods,procedures, modules and components may not have been described in detailso as not to unnecessarily obscure aspects of the embodiments.

The embodiments include a method, apparatus and system for colorquantification under various lighting conditions. The colorquantification can be used to detect and quantify specificconcentrations of biological analytes indicated by a diagnostic testdevice.

Principle of Illuminance Learning Algorithm

FIG. 12 shows a model for estimating illuminance (referred to as anilluminance model). The illuminance model is a regression model that isinitially trained in a learning or training phase/mode. Once trained,the illuminance model in a testing mode can estimate the illuminance ofa scene where digital photographs are captured.

FIG. 11 illustrates the illuminance model in the learning or trainingphase/mode to determine a best set of m parameters X₁ through X_(m) of aparameter vector X for estimating illuminance. During the learning phasefor the illuminance model, a learning/training data set of q colorsamples is input to the illuminance model. A sensor of the digitalcamera captures the color samples and forms m sensor features F_(im)1203 for each sample i. Accordingly, each sample i can be described by aline or row vector M_(i) as follows:M _(i)=[F _(i1) F _(i2) . . . F _(im)]

The vector M_(i) has m features and represents the sensory input 1203provided to the regression model 1201. There are q row vectors M_(i),one for each sample.

In the learning/training phase, there is an expected outcome to theilluminance, illuminance, 1204 for each sample i. During this learningphase, the variable Illuminance_(i) 1204 is recorded along with themeasurements of the sensor features F_(im) 1203. The recordedilluminance, 1204 for each sample i forms an illuminance matrix I. Themeasurements of the sensor features F_(im) 1203 for each sample i formsa sensor input matrix F.

The goal of learning phase is to find the best estimate of vector X 1202for the regression algorithm 1201. The vector X 1202 is the best set ofm parameters X_(i) through X_(m) for estimating illuminance. Intranspose matrix form, the parameter vector X 1202 isX ^(T)=[X ₁ X ₂ X ₃ . . . X _(m)]

The regression algorithm 1201 multiplies the sensor input matrix F withthe parameter matrix X to determine the illuminance matrix I. Inexpanded form the regression algorithm 1201 performs the matrixoperation of

${\begin{bmatrix}F_{11} & F_{12} & \ldots & F_{1\; m} \\F_{21} & F_{22} & \ldots & F_{2\; m} \\\vdots & \vdots & \ddots & \vdots \\F_{q\; 1} & F_{q\; 2} & \ldots & F_{q\; m}\end{bmatrix}*\begin{bmatrix}X_{1} \\X_{2} \\X_{3} \\\vdots \\X_{m}\end{bmatrix}} = \begin{bmatrix}{Illuminance}_{1} \\{Illuminance}_{2} \\\vdots \\{Illuminance}_{q}\end{bmatrix}$

Knowing each value of the F matrix and each value of the illuminancematrix I, the m parameters X₁ through X_(m) for the parameter matrix Xcan be determined. With the parameter matrix X determined, theilluminance model can then be used in a test mode/phase to determineilluminance.

FIG. 12 illustrates the test or measurement phase of the illuminancemodel. It is desirable to determine the illuminance, under which animage of a test sample was captured. The sensor includes a plurality ofm sensor features Feature₁ through Feature_(M) 1301. The sensor capturesthe test sample generating a measurement vector M 1302 with measurementfeatures F1 through Fm. The vector M is represented in matrix form asfollows:M=[F ₁ F ₂ . . . F _(m)]

In the illuminance model, the vector M vector 1302 as well as thepreviously learned parameter X vector 1202, are provided to theregression algorithm 1201. In transpose form, the vector X vector 1202is represented as followsX ^(T)=[X ₁ X ₂ X ₃ . . . X _(m)]

With the measurement vector M and the parameter vector X, theilluminance model can predict the illuminance from the following matrixequationM*X=Illuminance

While the illuminance model is shown and described using a linearregression and linear equations, a non-linear regression and non-linearequations may be used to estimate illuminance.

Applications

Measuring with high accuracy the illuminance of a scene with a portableelectronic device is one application. Camera metadata of a photographare insufficient to accurately predict illuminance. In vector form,camera metadata generally provides:M _(Meta i)=[ExposureTime_(i) ShutterSpeed_(i) FNumber_(i) Aperture_(i)Brightness_(i) IsoSpeedRating_(i)].

Referring now to FIG. 14, images of predetermined colored objects withdifferent colors may be used to gain accuracy of lux measurements with adigital camera, some of which are commonly carried by users. For abaseline, the different colors of the predetermined colored object canbe predetermined colors under known lighting conditions with knownilluminances. Digital photos or images of the predetermined coloredobject can then be captured under unknown lighting conditions andunknown illuminances. A measure of illuminance for the digital photoscaptured under unknown lighting conditions can then be determined. Themeasure of illuminance can be used to more accurately extract colors andfeatures from portions of a digital photograph or a colored object. Itmay be desirable to examine, authenticate, and/or validate the coloredobject or images of colored objects, persons, or parts thereof within adigital photograph. The analysis of the biological samples, such asblood and urine for example, is of great interest to determineconcentrations of analytes as a function of color.

The predetermined colored object may be a colored seal 1401 for example.The predetermined colored object may be a colored bank note 1402 oftenfound in purses, wallets, or pockets. The predetermined colored objectmay be a color identification (ID) badge or pass 1403 often worn onclothes or carried by a user. The predetermined colored object may becolor jewelry 1404 that is worn by a user. The predetermined coloredobject may also be a special object dedicated to predeterminedapplications, such as a diagnostic instrument 110 or medical device. Toimprove lux measurements with a portable electronic device including acamera, it is desirable that the predetermined colored object containsmultiple regions with diverse colors.

Referring now to FIG. 15, a state driver license 1405 is illustrated.The driver license 1405 may be another predetermined colored object thatcan be used to improve the accuracy of lux measurements obtained fromdigital photos taken with a digital camera. The state driver license1405 includes a plurality of regions with diverse colors. An example ofsuch regions is illustrated by points in the layout 1500 of the license,and points in the one or more pictures 1501-1502 of the driver license.

In this case, a template-based algorithm with predetermined knowledge ofthe drivers license extracts colors of features F₁, F₂, . . . . F_(m)corresponding to the driver's license layout 1500. The color featurescan be arranged into a vector Mtemp as follows:Mtemp=[F ₁ F ₂ . . . F _(m)]

A digital image of the license may be scanned to detect a feature andthen extract the color of that feature. For example, a pink hued headerbackground may be detected and the pink color extracted from the digitalphotograph. Clouds in the pink hued header background may be detectedand the dark pink color extracted from the digital photograph. A bluestate name in the header background may be scanned and detected. Theblue color in the state name may be extracted for analysis. The color ofthe background detected in the driver license may be extracted.

A facial recognition algorithm extracts colors of the features in thedriver's picture 1501. One feature in the picture, for example, may bethe color of the eyes. The algorithm may scan for pictures of faces anddetect the eyes of each face in a picture. The color of the eyes may beextracted. The color of the face/skin detected at different locations inthe picture may be extracted. The color of hair at different locationsdetected in the picture may be extracted. The color of the backgrounddetected in the picture 1501 of the driver license may be extracted. Avector Mpic1 with these features F₁′ through F_(n)′ from the driverpicture 1501 can be formedMpic1=[F′ ₁ F′ ₂ . . . F′ _(n)]

If there is another picture, such as a second driver picture 1502, thesame algorithm may be applied to extract color of features from thepicture. Another vector Mpic2 with color of features F₁″ through F₁″extracted from the driver picture 1502 can be formedMpic2=[F″ ₁ F″ ₂ . . . F″ ₁]

The simplest feature extraction analyzes the region and extracts itsdominant color. For example, a dominant color may be defined as themedian RGB color of the region.

A complete set of features extracted from the predetermined object canbe formed into a merged vector M, including the metadata of the digitalphoto, as follows:M=[M _(Meta) M _(temp) M _(pic1) M _(pic2)]

The features and colors of the diagnostic instrument 110 or medicaldevice can be similarly extracted and merged into a vector for analysis.An ID tag on the instrument may provide the locations of the features tobe extracted from the diagnostic instrument.

FIGS. 1A-1B illustrates a urine analysis test strip 101 and a urineanalysis test paddle 110, for example. Accurate inference of colors ofpixels in images captured by digital cameras has many practicalapplications. One such application is the analysis of urine with teststrips and test paddles, where colors of chemical agents changedepending on concentration of target substances in the urine samples.Other body fluids, such as blood, may be similarly analyzed with colorrepresenting the concentration of target substances in the biologicalsamples.

Learning Illuminance

Reference is now made to FIG. 9. During the learning phase, numerous(e.g., hundreds) images are generated for each illuminance [1 . . . n],resulting in several (e.g., thousands) overall tests q. After all testsare performed, the learning algorithm determines the mathematical modelcapable of best predicting Illuminance_(x), based on vector M_(x).

The illuminance model is formed with a regression model, which iscapable to classify the illuminance into n bins of illuminance values inresponse to the observation vector M_(i). In the simple case of a linearregression, the illuminance model is trained with the parameter vectorX, the transpose of which is as follows:X ^(T)=[X ₁ X ₂ X ₃ . . . X _(m)]In the case of non-linear regression, the illuminance model isgeneralized to a set of matrices.Testing the Illuminance

Referring now to FIG. 10, the measured vector M determined from thecaptured picture is entered to test the illuminance model underoperation. The illuminance switch mathematical model 923 is used withthe parameter vector X that was learned in the learning mode 922. Thus,for a linear regression model, the test/operation for illuminance 1101is the simple matrix multiplication:M*X=Illuminance

The matrix multiplication allows classifying the illuminance result intoone of the n illuminances 1101A-1101N that were used in training thesystem.

For quality assurance purposes, an illuminance sensor 1001 may be usedto obtain a measured value of illuminance. At process block 1102, aquality assurance test may be performed by comparing the measured valueof illuminance from the sensor 1001 to the estimated illuminance 1101predicted by the illuminance model. With this comparison, the accuracyof the learning algorithm and illumination model can be assessed bycalculating an average error or a standard deviation from the actualmeasurements. However, in practice, the illuminance sensor 1001 istypically unavailable.

Data Mining

Machine learning techniques are used to look for cloud separations inpoints of a database to train a model to discriminate an illuminancelevel over n illuminance levels. Data mining methods are used on thedata captured when taking a plurality of pictures. In one embodiment,data mining techniques analyze all the dimensions of the captured vectorM and the dimensions of the metadata vector M_(Meta) to extractparameters for the mathematical illuminance model to determineilluminance in a digital photo when captured.

Referring now to FIG. 13, a server 1301 may be used to data mine adatabase of a plurality of digital photographs. The server 1301 includesa processor 1311, a memory 1312, and one or more storage devices (e.g.,hard disk drives) to store instructions of database software and datamining software 1315, and the database 1313 of digital photographs. Theprocessor 1311 executes instructions of the database software and datamining software 1315 to perform the data mining functional processes.

Each of a plurality of portable devices 1350A-1350N (collectively 1350)include digital cameras 1352A-1352N to capture digital photos of a scenethat may include a reference color bar and/or chemical test pads. Someof the portable devices may be similar. Other portable devices maydiffer. The database may take the make and type of digital camera intoconsideration.

The digital photos may be captured under known differing illuminancevalues representing a plurality of different lighting conditions1360A-1360M, from bright sunlight 1360A to dark shadows 1360M, forexample. Each portable device 1350 and its camera 1352 may becharacterized under every light condition and illuminance value bycapturing a digital photo of the same scene. In this manner, the datamining software and server can determine parameters to use for the modelto discriminate illuminance in digital photos taken under unknownlighting conditions and illuminance.

The model and the parameters for the model may be stored into a storagedevice (e.g.) of a portable device 1350A-1350N for use by a processor,such as the computer readable media 208 for use by the processor 205 ofthe portable electronic device 220 shown in FIG. 2A. The portable device1350A-1350N can then capture digital images and discriminate theilluminance under unknown lighting conditions with the model and itsparameters. The processor 205 may have additional software instructionsin a software application to analyze chemical test pads and determinetitrations or concentrations of an analyte in a biological sampleapplied thereto.

The physical nature of illuminance and the supporting data imposesconstraints on the data mining method and the underlying mathematicalmodel. Illuminance is typically a continuous measurement. During imageacquisition it is desirable that illuminance be constant or vary slowly.It is desirable that data mining methods be noise-resistant in thiscontinuous environment. The severity of errors may be quantified by thedifference between the predicted illuminance and the measuredilluminance. The noise resistance and severity of errors can leadtowards mathematical methods forming convex partitions in the learningspace.

Yet another constraint on the data mining method stems from the use ofthe training of the illuminance model in an application running onmobile devices. With the application being downloaded, it is desirablethat the mathematical model be compact to reduce the download size andthe download time. Furthermore, it is desirable that the illuminancemodel use modest processing resources during the test phase of thealgorithm since it is run on mobile devices having constrained resourcesin both processing power and memory. Moreover, it is desirable tominimize processing time to conserve energy of the mobile device.

Several data mining methods were tested, amongst them all linearmethods, including linear regression and linear discriminant analysis,failed to achieve a great accuracy. A naïve Bayesian classificationmethod also failed to reach great accuracy as many parameters of themodel were linked therefore violating a Bayesian hypothesis.

Other data mining methods performed well. A tree classification methodperformed well, providing a compact mathematical model. However, noiseresistance was lacking Additionally, the tree classification methodresulted in convex illuminance clusters in the data mining space.

The K nearest neighbor data mining method performed well in terms ofaccuracy. However, it resulted in convex illuminance clusters in thedata mining space. Moreover, its resulting mathematical model is verylarge due to an enumeration of all the learning samples.

Diagnostic Instruments

The invention relates generally to systems and methods for detecting thepresence or absence of a variety of analytes in a fluid sample used witha diagnostic instrument. Diagnostic test results are determined by imageanalysis of a digital image of the diagnostic instrument. The analysisis performed without the use of any controlling light, such as a flashof light from a camera light source in flash mode, a constant light fromthe camera light source in torch mode, or any other/additionalcontrolled source light. Rather, the measure of illuminance isdetermined from existing sensors within digital cameras of personalelectronic devices. Moreover, a measure of illuminance can be determinedby the embodiments without any external controlled lighting, such as aflash, a torch, or a light emitting diode (LED).

Reagent dipsticks and immunoassays have been used in medical clinics fordecades in connection with methods for rapidly diagnosing healthconditions at the point of care. In a clinical environment, dipstickshave been used for the diagnosis of urinary tract infections,preeclampsia, proteinuria, dehydration, diabetes, internal bleeding andliver problems. As is known, dipsticks are laminated sheets of papercontaining reagents that change color when exposed to an analytesolution. Each reagent test pad on the dipstick is chemically treatedwith a compound that is known to change color in the presence ofparticular reactants. For example in the context of urinalysis, thedipstick will typically include reagent pads for detecting or measuringanalytes present in a biological sample such as urine or blood,including glucose, bilirubin, ketones, specific gravity, blood type,blood concentration, acidity (pH), protein, urobilirubin (urobilinogen),nitrite (nitrates), leukocytes, microalbumin and creatinin. Themagnitude of the color change of the reagent test pad is proportional tothe analyte concentration in the patient fluid.

Referring now to FIG. 1A, an example dipstick 101 and an example colorreference chart or color interpretation chart 103 are shown. The colorinterpretation chart 103 is used to make a direct color comparison witha dipstick 101 that has been exposed to a body fluid, such as urine orblood for example. The color interpretation chart 103 is related to thedipstick 101, often being assembled together by the same manufacturer,and thus may be referred to as a Manufacturer Interpretation Color Chart(MICC) 103.

Dipsticks 101 are typically interpreted with a user's naked eye bycomparing the test strip 101 against a color reference chart 103. Adipstick reagent color 102 is compared to a set of possible colors 104corresponding to possible concentrations/titrations/quantity of thetested reagent. However, such color comparison can cause user confusionand error, for several reasons including changes in ambient lighting,and that a significant portion of the population has impaired colorvision.

Automatic methods and apparatus for interpreting test results ofdipsticks and immunoassays, which have been exposed to a samplesolution, are known in the art. One of the approaches is to build adedicated machine taking the samples into a controlled environment wheresensors read the dipsticks illuminated by known light sources. Anotherapproach is to use a camera to capture side by side the dipsticks and aManufacturer Interpretation Color Chart (MICC) to make a direct colorcomparison, since test and result interpretation charts are seen underthe same lighting. Our approach described in '842 application puts theleast constraints on the end-user, as it configures and automaticallycalibrates the digital image to spectrally correct for any colordeficiencies, artifacts, or other ambiguities.

Some embodiments of the invention are drawn to diagnostic instruments,systems and methods of use thereof for testing of a patient fluidsample, which can be used either in clinical settings or for home use.More particularly, embodiments of the invention relate to theperformance of color-based reaction testing of biological material in anautomatically calibrated environment. In some embodiments, the inventionis implemented as an application running on a portable electronicdevice, such as a cell phone, tablet PC, computer, laptop, ahead-mounted display like ‘glasses’ or other dedicated electronicdevice. The method has been designed to minimize user contact andmanipulations of biologically soiled samples.

With reference to FIG. 1B, a diagnostic instrument 110 is introduced. Itis made of a paddle 111 that is populated with the dipstick reagentcolor 112 and a unique identification tag 114. In response to the tag114, a map links the reagent types 112 to locations on the paddle 111.The diagnostic instrument could be interpreted manually with a MICC 103.However, with a digital camera of a portable electronic device, colorsof exposed reagent types 112 can be automatically interpreted with colorquantification methods and algorithms.

In FIG. 2A, a system 200 for diagnosing a patient condition includes thediagnostic instrument 110 and a portable electronic device 220.Generally, the portable electronic device 220 of the system 200 acquiresimage data of diagnostic instrument 110. The portable electronic device220 is used for evaluating, analyzing, processing, and/or presenting theimage data of diagnostic instrument 110. The system 200 may be used inany type of medical analytical/diagnostic setting, including a medicalclinic, an off-site laboratory, or in home without medical supervision.

In certain non-limiting embodiments, the portable electronic device 220includes a camera sensor 202, a flash or lighting device 203 (e.g., oneor more light emitting diodes or a xenon flash bulb), a processor 205,computer-readable media 208, a reference tag reader 204, a visualdisplay device 201, a digital image analyzer 206, a data transmitter207, a date entry device 209, and a timer 210.

The camera sensor 202 obtains the digital image of the diagnosticinstrument 110. The processor 205 is configured to execute programinstructions stored on computer-readable media 208 associated with theportable electronic device 220.

The computer-readable media 208 may include computer storage media, suchas media implemented in any method or technology for storage ofinformation, such as computer-readable instructions, data structures,program modules, or other data, random access memory (RAM), read onlymemory (ROM), electrically erasable programmable read only memory(EEPROM), flash memory, or other memory technology, CD-ROM, digitalversatile disks (DVDs), or other optical disk storage, magneticcassettes, magnetic tape, magnetic disk storage, or other magneticstorage devices, or any other medium which can be used to store thedesired information and which can be accessed by an electronic device,such as the portable electronic device 220 and its processor 205.

In addition to storing the program for controlling functions of theportable electronic device 220, the computer-readable media 208 may alsostore data including one or more tables of colors for one or more MICCcharts 103 that may be used for comparison to determine test results ofthe diagnostic instrument 110.

The data transmitter 207 is for transmission of data and informationfrom the portable electronic device 220 to an external electronicdevice, a computer or server in a computer network, and/or a digitalstorage device, collectively referred to as a network environment (alsoreferred to as “the cloud”) 211. Once the data is provided to thenetwork environment 211, it may be made available to third partyapplications 212 and used by caregivers, doctors, payment organizations,insurance and health maintenance organizations, pharmacists, or publichealth organizations.

In FIG. 2B, a smartphone or tablet computer 220 may be used with thediagnostic paddle 110 to obtain results. The smartphone or tabletcomputer 220 includes a camera 202 and a display device 201 that may beused to obtain and display results from the diagnostic paddle 110. Thedisplay device 201 may provide a test paddle (TP) display area 251 todisplay the test paddle 110 and a user interface (UI) display area 252to display instructions and results to the user. The smartphone ortablet computer 220 further includes a processor 205 and a memory 208 tostore instructions for execution by the processor. The instructions maybe software that provide the user interface in the UI display area 252,capture images for display in the display area 251, and perform thealgorithms and the methods described herein to obtain the test results.

Referring back to FIG. 1B, the diagnostic instrument 110 is made of apaddle 111, an ID tag 114 and a set of chemical reaction pads orchemical test pads (CTP) 112. The ID tag 114 allows a uniqueidentification and storage of information such as lot number, expirationdate, legal mentions etc. associated with the diagnostic instrument. Theset of chemical reaction test pads (CTP) 112 perform the tests of thediagnostic instrument. The position of each of the CTP 112 correspondsto a precise test of a biological sample, such as glucose, bilirubin,ketones, specific gravity, blood, pH, etc.

Reference is now made to FIGS. 3A-3B. In FIG. 3A, a flow chart of amethod for obtaining diagnostic results from the diagnostic instrument110 is shown. The process begins at block 301.

At process block 301, the user initiates the method by pressing a buttonon the display screen 201 of the portable electronic device 220 so thatthe processor 205 executes a program. The user exposes the set of CTPs112 of the diagnostic instrument 110 to a biological sample. Theanalytes contained in the biological sample start a chemical reaction inthe CTPs 112 triggering their color change. In certain embodiments, atimer 210 of the portable device 220 is started when the diagnosticinstrument 110 is exposed to the biological sample. After apredetermined time passes as may be measured by a clock or timer, suchas timer 210 of the device 220, the portable electronic device 220prompts the user to capture one or more digital images of the diagnosticinstrument 110.

At process block 302, a digital image of the diagnostic instrument 110is captured using the camera sensor of a portable electronic device 220.The captured digital image includes at least the CTPs, and the ID tag ofthe diagnostic instrument. If included on the diagnostic instrument, thecaptured digital image may also include a reference color bar.

At process block 303, the position and color of one or more of the CTP102 are then extracted from the captured digital image of the diagnosticinstrument 110. The position of each CTP 112 is then mapped to the typeof test it performs (e.g. glucose, acidity (pH)) according to apredetermined map of the set of test pads. Each of these tests can beinterpreted by comparing the CTP color to a line of reference colors 104in the Manufacturer Interpretation Color Chart 103. A line of referencecolors 104 represents the color trajectory in the RGB space that aspecific chemical test pad follows when submitted to all possibleconcentrations/titrations/quantity in its detection range. U.S. patentapplication No. 61/680,842 discloses a continuous trajectory linking allpoints of the discrete MICC color line 104 of a given analyte toincrease accuracy of color quantification. The color trajectories 311are constructed once, during the calibration procedure of theinstrument. The trajectories are formed under known lighting conditions,typically under a lightbox. The color trajectories 311 are stored aslook up tables into memory of the device 220 and recalled by theprocessor to interpret the color of a CTP 112.

At process block 304, under the same known lighting conditions socorrection is unnecessary, a color interpretation of the color of theCTP 112 can be made with a simple look-up mechanism by the processor.The look up mechanism maps the values of the camera-captured CTP colorsto the nearest point in the corresponding color trajectory 311 of thegiven analyte of the CTP. The results or titration is directly read outfrom the trajectory given the camera captured CTP color. There is aone-to-one mapping between a point on the color trajectory 311 and atitration or concentration of analyte in a biological sample.

At process block 305, the result and the titration are reported to auser by the device 220.

The processes 303-305 are repeated for each CTP numbering from i equalto one to a maximum CTP number N. A result and titration (i) is providedfor each CTP (i). For the exemplary diagnostic instrument 110, twelveCTPs are analyzed for test results and titration. However, otherimplementations of diagnostic device 110 may have a different number ofCTPs 112.

At process block 399, after all the CTPs of a diagnostic instrument havebeen color quantified and interpreted to a quantity (e.g.,concentration/titration), the process ends.

The color quantification method shown in FIG. 3B works well under knowncontrolled lighting conditions and illuminance, such as a light box at1600 Lux for example. Additional details of a color quantificationmethod are described in U.S. patent application No. 61/680,842 that isincorporated herein by reference.

However, it is desirable for a color quantification method and system tofunction over a large range of lighting conditions. Color correctionscan be automatically introduced to normalize the extracted colors fromthe CTPs to the lighting conditions and then comparisons can beperformed in this automatically calibrated color space. Experiments haveshown that illuminance influences the color trajectories 311 of analytesbeyond the effect of the color corrections.

FIG. 4 illustrates a plurality of MICC color trajectories 401-405 whenobserving the same CTP (e.g., a Glucose CTP) under several differentilluminance conditions. Under bright lighting conditions of 1600 Lux,the color trajectory 405 shows the different colors for the given CTPover the varying concentrations/titrations. For example, the CTP mayhave a zero analyte concentration at an RGB of approximately 95,184,172for the illuminance of 1600 lux on trajectory 405. The CTP may have amaximum analyte concentration at an RGB of approximately 86, 29, −37 forthe illuminance of 1600 lux on trajectory 405. The color trajectorycurve 405 provides good system performance at an illuminance of 1600Lux, such as may be expected in lighting conditions of an office or astudio, for example. However in dimmer lighting conditions, the 1600 luxcolor trajectory 405 is not optimal to use in making color comparisonswith an extracted color from a CTP.

Additional color trajectories 401-404 were experimentally formed withlower illuminance to extend the operating range of the device 220 to lowlighting conditions, such as may be found in corridors, bathrooms,living rooms, kitchens, etc. Color trajectories 401, 402, 403, and 404were formed corresponding to analyte (e.g., glucose)concentrations/titrations observed under illuminances of 60, 160, 260,and 550 lux respectively. Alternatively, the 1600 lux color trajectorymay be mapped to the color trajectories 401-404 respectively forilluminances of 60, 160, 260, and 550 lux by a mapping algorithm.

While only five color trajectories corresponding to analyte (e.g.,glucose) concentrations extracted under five different illuminances(e.g., 60, 160, 260, 550, and 1600 lux) are shown in FIG. 4, theembodiments are not to be limited to this number of illuminances andcolor trajectories. Additional color trajectories for each reagent andtest may be formed for other illuminances, where each illuminance levelgenerates its own color trajectory curve. Generally, there is a variancein the color of a CTP under different lighting conditions that is takeninto account to acquire a more accurate titration level.

General Color Quantification Algorithm

The method of color quantification shown in FIG. 3A may be adapted to alarge range of lighting conditions that may be captured usingoff-the-shelf cellular phone cameras.

In FIG. 5, a process 511 is added over those shown in FIG. 3A. Atprocess 511, the illuminance of the diagnostic instrument 110 isestimated based on its acquired picture. This estimated value ofilluminance is used to select the proper color trajectory from theplurality of color trajectories 304A-304N to provide accurate colorinterpretation results.

In one embodiment, there may be five color trajectories 304A-304Ecorresponding to illuminances of 60, 160, 260, 550 and 1600 Lux. Themultiple color trajectories 304A-304E represent the various lightingconditions that may be found inside dwellings, ranging from 60 lux for apoorly lighted environment (e.g., a corridor/closet) to 1600 lux for awell lighted environment (e.g., an office with one or more windows).

After one of the trajectory curves is selected in response to theestimated illuminance, the RGB components of the color are interpretedon the trajectory curve into an analyte concentration/titration. Theinterpretation is performed by a look up table with the RGB componentsbeing the entry points into the table.

At process 305, the results and titration of each CTP are presented tothe user. The determined illuminance level may also be presented to theuser as part of the results and titration.

Processes 302, 303, and 306 are similar to those described withreference to FIG. 3A.

The processes of FIG. 5 are repeated for each CTP of the diagnosticinstrument (e.g., diagnostic instrument 600 shown in FIG. 6). After allCTPs of the diagnostic instrument have been considered, the process goesto process block 599 and ends.

Estimating illuminance for the accuracy and reliability of a medicaldevice is challenging, especially when using built-in digital cameras ofportable electronic devices. Most digital cameras prohibit direct accessto their light sensors. Additionally, raw illuminance measurements bythe light sensors are manufacturer dependent and largely unspecified.Thus, it is difficult to use the raw illuminance measurements from thelight sensors over a wide range of families of devices from differentmanufacturers. Moreover, manufacturers may regularly announce newdevices 220 during the year, such that any luxmeter software applicationmay need regular updates to follow the evolution of new devices.Accordingly it may be desirable to avoid using the light sensors thatprovide raw illuminance measurements. Instead, the embodiments canestimate illuminance based on the captured image content and basicmetadata collected by cameras. Accordingly, new visual clues areintroduced into the diagnostic instrument 110 to assist in theestimation of illuminance.

Referring now to FIG. 6, a diagnostic instrument 600 is shown with suchnew visual clues. The diagnostic instrument 600, also referred to as atest paddle, includes a color reference bar or reference color bar (RCB)601 adjacent the reagent or chemical test pads 112. The reference colorbar (RCB) 601 provides visual color clues on the test paddle 600 toassist in the estimation of illuminance, such as normalizing orcorrecting colors of the chemical test pads, under different lightingconditions. In some embodiments, the colors of reference color bar (RCB)601 represent the nominal colors of the CTPs at one or more targetconcentrations in a biological sample.

The test paddle 600 further comprises a substrate 610 with a handle 614at one end and a blade 616 at an opposite end. The set 620 of reagenttest pads 112 are mounted to the blade portion 616 of the substrate 610with an adhesive. The substrate may be plastic or a heavy duty paper,such as cardboard.

In one embodiment, the test paddle 600 has twelve reagent pads 112. Thetwelve reagent pads 112, also referred to as chemical test pads (CTP) orsimply test pads, are positioned near the blade of the paddle 600. Inthis exemplary embodiment, the CTPs 112 are arranged into an array ofthree rows (112-1 x through 112-3 x) and four columns (112-xA through112-xD). Different arrays of CTPs may be used with differing numbers ofCTPs on the test paddle.

Each CTP 112 may be treated with a chemical compound (a reagent)specifically selected to react with a specific analyte within abiological sample. Other CTPs may be treated with another chemicalcompound (reagent) to determine other factors about the biologicalsample, such as acidity (PH) or specific gravity for example.

Between the handle and the reagent test pads 112 is the reference colorbar (RCB) 601. The test paddle 600 may further include a matrix bar codeor a two-dimensional bar code 114. The test paddle 600 may furtherinclude an opening 615 capable of hosting an additional test, such as apregnancy test.

The reference color bar 601 includes a plurality of color samples in aside-by-side linear arrangement. For example, the reference color bar601 may include color samples for one or more of the following colors:Cyan, Magenta, Yellow, Key (black), Gray, White, Red, Green, Blue. Thesample colors correspond with common color spaces, such asRed-Green-Blue or Cyan-Magenta-Yellow-Key (black). The known colorsamples may be shaped as color squares along the reference color bar oralternatively other two dimensional shapes such as color circles, colorovals, or color rectangles.

The reference color bar 601 is used for image processing. The RCB 601may be used to calibrate or normalize a digital image of the diagnosticinstrument to improve the quality and accuracy of color analysis.Additionally, the reference color bar 601 may be used to determine theilluminance. The reference color bar (RCB) 601 may be used toautomatically correct the captured colors of the different reagent testpads 112, prior to a comparison with a set of color calibration curvesto determine analyte concentration in the test sample. The new visualclues provided by the reference color bar 601 on the test paddle 600lead to an improved flow chart for color quantification.

The matrix bar code or two-dimensional bar code 111 may also referred toas a quick response (QR) code 114. The quick response (QR) code 114 canprovide a unique identification to automatically identify the testpaddle 600. The QR code 114 may be configured to contain certainidentification information about the test paddle 600, such as a list ofthe analytes that are being tested, expiration date of the paddle 600,the conditions that are being tested, and other identifying information.The information may be printed directly on the unique identification orencrypted within the QR code 114.

Alternatively, the QR code 114 may be associated with information storedelsewhere, such as is the case with bar codes or other near-fieldcommunication codes. The identification information may be used in avalidation processes to ensure the diagnostic test paddle 600 issuitable for the tests being performed and to ensure that it is safe touse, in good working condition, or to resolve other issues which mayimpact quality and reliability of the test results.

Referring now to FIG. 7, a flow chart illustrates a method of colorquantification including the processing of the image of the referencecolor bar chart (RCB). The method starts at block 701 and goes toprocess block 302.

Processes 302, 303, 305, and 306 are similar to those described withreference to FIG. 3A. Process 511 and selection of one of thetrajectories 304A-304N is similar to that described with reference toFIG. 5. The method illustrated by FIG. 7 introduces process blocks 711and 702 to that of FIG. 5. The processes of FIG. 7 are repeated for eachCTP of the diagnostic instrument. After all CTPs have been considered,the process goes to process block 799 and ends.

At process block 711, RGB color values for the reference colors of thereference color bar 601 are extracted from the captured picture of thediagnostic instrument 600. The reference color bar values are used tomake two kinds of corrections at process blocks 702 and 511.

At process block 702, a color normalization process occurs. The colornormalization process transforms the perceived colors from each CTP intoa normalized color space. This operation calculates the inversetransform to be applied to the perceived RCB colors so that these colorsappear as seen under a normalized lighting environment, e.g. D65. Thisinverse transform is then applied to the perceived RGB colors of the CTPto determine their normalized color. The normalized RGB colors for eachCTP are used to determine the concentration/titration using a selectedcolor trajectory. The normalization process is further described in U.S.Patent Application No. 61/973,208 and incorporated herein by reference.

At process block 511, an illuminance correction is made with thereference colors of the reference color bar. The RCB colors of thereference color bar are used in the illuminance switch process 511, inconjunction with other parameters like the camera metadata and CTPcolors, to make a high precision luxmeter determination of theilluminance of the paddle.

Dimension of the Illuminance Switch Problem Space

Camera metadata may be some of the parameters that may be used by theilluminance model. Camera metadata for photograph i may be expressed invector form by an M_(Meta) vector (meta vector) as follows:M _(Meta i)[ExposureTime_(i) ShutterSpeed_(i) FNumber_(i) Aperture_(i)Brightness_(i) IsoSpeedRating_(i)]

Camera metadata of a photograph are insufficient to accurately measureilluminance. Additional data that extends the model is used toaccurately measure illuminance of a scene. The referenced color bar(RCB) 601 is introduced into the captured scene of the photograph tomore accurately measure illuminance. The test paddle 600 furtherincludes the chemical test pads 112 that are captured in the photographthat may also be used to more accurately measure illuminance.

For the reference color bar (RCB) 601, a new observation vectorcorresponding to the RGB colors of the sample colors—Cyan, Magenta,Yellow, Key (black), Gray, White, Red, Green, Blue—of the RCB 601 isintroduced and used in the model. The new observation vectorM_(percRCB i) has a dimension 27 (3 RGB×9 colors) and is as follows:M _(percRCBi)=[pCyan_(Ri) Cyan_(Gi) pCyan_(Bi) . . . pBlue_(Ri)Blue_(Gi) pBlue_(Bi)]The observation vector of the reference color bar may also be normalizedand used in the model. A normalized RCB adds another line vectorM_(normRCB i) (normalized reference color bar vector) into the modelwith the same dimension 27 as follows:M _(normRCBi)=[nCyan_(Ri) nCyan_(Gi) nCyan_(Bi) . . . nBlue_(Ri)nBlue_(Gi) nBlue_(Bi)]While the new observation vector mentioned above is based on known RGBsample colors, different numbers and combinations of known referencecolors may be used. In one embodiment, the reference colors of the RCBare nominal colors of each CTP at one or more, or all predeterminedtarget concentrations.

The observed or captured scene in the digital photograph includes theCTPs 112 of the test paddle 600. In one embodiment there are twelve CTPs112, CTP1 through CTP12, each of which is captured with the three red,green, and blue (RGB) color values. With twelve CTPs, the correspondingline vector M_(percCTPi) that is captured for a test paddle has adimension 36:M _(percCTPi)=[pCTP1_(Ri) pCTP1_(Gi) pCTP1_(Bi) . . . pCTP12_(Ri)pCTP12_(Gi) pCTP12_(Bi)]The measured or observation vector of the CTPs may also be normalizedand used in the model. A normalized CTP vector adds another line vectorM_(normCTP i) into the model with the same dimension 36:M _(normCTPi)=[nCTP1_(Ri) nCTP1_(Gi) nCTP1_(Bi) . . . nCTP12_(Ri)nCTP12_(Gi) nCTP12_(Bi)]

The reference color bar RCB vectors and the CTP vectors can be combinedtogether into a single line vector. For a given photograph i, the RCBand CTP vectors combine into a single M_(RCBCTP i) vector of dimension126:M _(RCBCTPi)=[M _(percRCBi) M _(normRCBi) M _(percCTPi) M _(normCTPi)]

Despite the high dimension of elements in the M_(RCBCTP) vector,determining illuminance of a test paddle is still a challenge. Variablesin the M_(RCBCTP) vector were analyzed to determine their use inclassifying images according to their illuminance. Of the 126 dimensionscorresponding to the RGB representations of all the points (CTP+RCB) inthe M_(RCBCTP) vector, there was no single dimension or combination ofdimensions found that could separate points and discriminate illuminancefrom different lighting conditions. Accordingly, the added sixdimensions of metadata are used to look for cloud separations in 132dimensions of problem space. With so many dimensions, machine learningtechniques are useful to look for cloud separations in points anddiscriminate illuminance.

Accordingly, data mining methods were introduced that work over all ofthe data captured when taking a plurality of pictures. In oneembodiment, data mining techniques include, but are not limited to, theone hundred twenty six (126) dimensions of the captured vectorM_(RCBCTP), and the six (6) dimensions of the metadata vector M_(Meta),for a total of 132 dimensions in accordance with one embodiment to forma vector M_(i):M _(i)=[M _(RCBCTPi) M _(Metai)]size(M _(i)=(1,132) in one embodimentLearning the Illuminance Switch

Reference is now made to FIGS. 8-9. FIG. 8 illustrates a flow chart ofthe learning method for the illuminance model. FIG. 9 is a block diagramillustrating how the method is used with the illuminance model in theexperimental learning setting. The learning environment performs testsunder controlled lighting, with illuminance 1005 set to one of the nbuckets of illuminance [1 . . . n]. The total number of tests q isdetermined by the acceptable error rate of the learning. For each test,an image is captured simultaneously by the camera 1002 and by theilluminance sensor 1001. The method begins at process block 901 and thengoes to process blocks 302 and 911 which may be performed concurrentlyin parallel.

At process block 911, the illuminance sensor 1001 measures theilluminance Illuminance_(i) at the diagnostic instrument 600. Themeasurement of illuminance is coupled to the process block 922.

At process block 302, a camera 1002 concurrently captures a photographor picture, a digital image of the diagnostic instrument 600 (alsoreferred to as a test paddle). The process then goes to process blocks303, 711, and 914 that can be concurrently performed in parallel.

At process block 303, the perceived colors of the CTPs are extractedfrom the captured photograph of the instrument 600.

At process block 711, the perceived colors of the RCB are extracted fromthe captured photograph of the instrument 600.

At process block 914, the camera metadata associated with the capturedphotograph of the instrument 600 is extracted from the capturedphotograph. The process may then go to process block 712.

At process block 712, the perceived CTP and the perceived RCB vectorsare normalized to form the corresponding normalized CTP vector andnormalized RCB vector. With the perceived RCB vector, the perceived CTPvector, the normalized RCB vector and the normalized CTP vector, theM_(RCBCTP) vector can be constructed which represents one line in theobservation vector M_(i) for the learning matrix. Both M_(i) andIlluminance_(i) are entered into the learning algorithm 922, in thefollowing system:

${\begin{bmatrix}M_{1} \\M_{2} \\\vdots \\M_{q}\end{bmatrix}*\begin{bmatrix}X_{1} \\X_{2} \\X_{3} \\\vdots \\X_{m}\end{bmatrix}} = \begin{bmatrix}{Illuminance}_{1} \\{Illuminance}_{2} \\\vdots \\{Illuminance}_{q}\end{bmatrix}$

During the learning phase, numerous (e.g., hundreds) images aregenerated for each illuminance [1 . . . n], resulting in several (e.g.,thousands) overall tests q. In one embodiment, the dimensionm=126+6=132. However, the illuminance model may be used with anydimension of vector. After all tests are performed, the learningalgorithm process 922 determines the mathematical model capable of bestpredicting Illuminance_(x), based on vector M_(x).

At process block 922, the illuminance model is formed with a regressionmodel that is capable of classifying the illuminance into n bins inresponse to the observation vector M_(i) In the simple case of a linearregression, the illuminance model is trained with the parameter vector Xthe transpose of which is as follows:X ^(T)=[X ₁ X ₂ X ₃ . . . X _(m)]In the case of non-linear regression, the illuminance model isgeneralized to a set of matrixes.Testing the Illuminance Switch

Reference is now made to FIG. 10. The parameter vector X for theilluminance model has been determined. During the test phase of theilluminance model, hundreds of images are generated by a light sourcewith a random illuminance value 1105, within the range of possibleilluminance values. Test images of the diagnostic instrument 600captured by the camera 1002 are treated independently. Each test imageis processed with the illuminance model similar to that of the learningphase and the method illustrated by FIG. 8 but for processes911,922,923. Usually the illuminance sensor 1001 and the captureilluminance process 911 shown in FIG. 8 are typically unavailable in anoperational mode. Instead of a learning mode, processes 922,923 functionin an operational mode. To test the operational mode, the illuminancesensor 1001 is used in the capture illuminance process 911 to comparewith the predicted illuminance 1101A-1101N.

At process 711, the perceived colors of the RCB are extracted from thepicture. At process 303, the perceived colors of the CTPs are extractedfrom the picture. At process 914, the camera metadata is extracted fromthe picture. At process 712, the perceived colors of the CTP and RCB arenormalized into normalized color CTP and normalized color RCB,respectively. The perceived and normal colors of the CTP and RCB arecombined together with the metadata into a measured vector M.

The measured vector M determined from the captured picture is entered totest the illuminance model under operation. The illuminance switchmathematical model 923 is used with the parameter vector X that waslearned in the learning mode with the illuminance switch learningalgorithm 922. Thus, for a linear regression model, the test/operationfor illuminance 1101 is a simple matrix multiplication as follows:M*X=Illuminance

The matrix multiplication allows classifying the illuminance resultsinto one of the n illuminances 1101A-1101N that were used in trainingthe system.

For quality assurance testing purposes, an external illuminance sensor1001 may be used in parallel with the camera 1002 to obtain a measuredvalue of illuminance when the captured photograph of the diagnosticinstrument is taken. At process block 1102, a quality assurance test maybe performed by comparing the measured value of illuminance from thesensor 1001 to the estimated illuminance 1101 predicted by theilluminance model. With this comparison, the accuracy of the learningalgorithm and illumination model can be assessed by calculating anaverage error of the measurements and its precision can be assessed bythe standard deviation of the measurements. However, in practice, theilluminance sensor 1001 is typically unavailable and the qualityassurance testing process 1102 is not performed.

Data Mining

As mentioned herein, machine learning techniques are used to look forcloud separations in points of a database (e.g., hyper planes separatingpoints in the database) to train the model to discriminate illuminanceover the n illuminance values to determine the best estimate ofilluminance. Data mining methods are used on the data captured whentaking a plurality of pictures. In one embodiment, data miningtechniques analyze the one hundred twenty six (126) dimensions of thecaptured vector M_(RCBCTP), and the six (6) dimensions of the metadatavector M_(Meta), to extract parameters for the mathematical illuminancemodel to discriminate illuminance in a digital photo from the nilluminance values that may be possible.

In one embodiment the data mining algorithm is a quadratic discriminantanalysis. It has precise results (e.g., in one case greater than 99.75%correct results), is noise resistant, and relies on a convexmathematical model. Moreover, it produces a compact mathematical modelthat is moderately processor (CPU) and memory intensive during the testphase, because it just performs matrix multiplication.

The preferred illuminance model transmits a squared matrix of size 132,in addition to 5 vectors of size 132, each of which represents thecentroid of an illuminance bucket.

Optionally an additional step of principle component analysis (PCA) canbe applied to the data mining algorithms to reduce the size of themathematical model to be transmitted to the application. In this case,the size of the preferred embodiment can be reduced from a dimension of132 to 20. The tradeoff is approximately a 1% loss of accuracy (>98.75%correct results) for a reduction of the mathematical model size by afactor of 36.

Other machine learning techniques may be used as well, such as supportvector machine (SVM), random forests, neural networks, deep beliefnetworks, and deep Boltzman machines.

CONCLUSION

When implemented in software, the elements of the embodiments of theinvention are essentially the code segments or instructions to performthe functional tasks described herein. The code segments or instructionsare executable by a processor and can be stored in a storage device or aprocessor readable storage medium, awaiting execution. The processorreadable storage medium may include any medium that can storeinformation. Examples of the processor readable storage medium includean electronic circuit, a semiconductor memory device, a read only memory(ROM), a flash memory, an erasable programmable read only memory(EPROM), a magnetic disk, a floppy diskette, a hard disk, an opticaldisk, a compact disk read only memory (CD-ROM), and a Blu-ray disk. Thecode segments or instructions may also be downloaded via computernetworks such as the Internet, Intranet, etc. and stored into a storagedevice or processor readable storage medium and executed by theprocessor.

While certain embodiments of the disclosure have been described, theseembodiments have been presented by way of example only, and are notintended to limit the scope of the disclosure. Various combinations andsub-combinations, and modifications as may be made, of the presentlydisclosed components and embodiments and aspects are contemplatedwhether or not specifically disclosed hereunder, to the extent and aswould be apparent to one of ordinary skill based upon review of thisdisclosure and in order to suit a particular intended purpose orapplication. Indeed, the novel methods, systems, and devices describedherein may be embodied in a variety of other forms. Furthermore, variousomissions, substitutions and changes in the form of the methods,systems, and devices described herein may be made without departing fromthe spirit of the disclosure. For example, certain features that aredescribed in this specification in the context of separateimplementations may also be implemented in combination in a singleimplementation. Conversely, various features that are described in thecontext of a single implementation may also be implemented in multipleimplementations, separately or in sub-combination.

Moreover, although features may be described above as acting in certaincombinations and even initially claimed as such, one or more featuresfrom a claimed combination may in some cases be excised from thecombination, and the claimed combination may be directed to asub-combination or variations of a sub-combination. Accordingly, theclaimed invention is to be limited only by patented claims that followbelow.

What is claimed is:
 1. A method comprising: exposing a plurality ofchemical test pads of a diagnostic instrument to a biological sample;waiting for a predetermined amount of time so that a chemical reactionat the plurality of chemical test pads with the biological sample can becompleted; capturing a digital color photograph of the diagnosticinstrument and the plurality of chemical test pads with a digitalcamera; extracting a color of each of the plurality of chemical testpads from the digital color photograph; mapping each extracted color toeach of the plurality of chemical test pads; looking up a colortrajectory for each of the plurality of chemical test pads; comparingthe extracted color of each of the plurality of chemical test pads witheach respective color trajectory for each chemical test pad; anddetermining a diagnosis of one or more tests of the biological sample inresponse to based on the comparing of the extracted color of each of theplurality of chemical test pads with each respective color trajectory ofeach chemical test pad.
 2. The method of claim 1, wherein one of the oneor more tests is for a concentration of an analyte in the biologicalsample.
 3. The method of claim 1, wherein the diagnostic instrumentincludes a reference color bar that is captured in the digital colorphotograph, and the method further comprises prior to the comparing,normalizing the extracted color of each of the plurality of chemicaltest pads using one or more colors of the reference color bar capturedin the digital color photograph.
 4. A diagnostic system for biologicalsamples, the diagnostic system comprising: a diagnostic instrumenthaving a plurality of chemical test pads to receive a biological sample;a portable electronic device including a digital camera to capture adigital image of the diagnostic instrument, a processor coupled thedigital camera to receive the digital image and metadata associated withthe digital image, and a storage device coupled to the processor, thestorage device to store instructions for execution by the processor thatwhen executed cause the processor to time a chemical reaction at theplurality of chemical test pads with the biological sample tocompletion; capture a digital color photograph of the diagnosticinstrument and the plurality of chemical test pads with the digitalcamera; extract a color of each of the plurality of chemical test padsfrom the digital color photograph; map each extracted color to each ofthe plurality of chemical test pads; look up a color trajectory for eachof the plurality of chemical test pads; compare the extracted color ofeach of the plurality of chemical test pads with each respective colortrajectory of each chemical test pad; and determine a diagnosis of oneor more tests of the biological sample based on the comparison of theextracted color of each of the plurality of chemical test pads with eachrespective color trajectory of each chemical test pad.
 5. The diagnosticsystem of claim 4, wherein one of the one or more tests for which adiagnosis is determined is a concentration of an analyte in thebiological sample.
 6. The diagnostic system of claim 4, wherein thediagnostic instrument further includes a reference color bar that iscaptured in the digital color photograph, and the storage device storesfurther instructions for execution by the processor that when executedfurther cause the processor to normalize the extracted color of each ofthe plurality of chemical test pads with one or more colors of thereference color bar captured in the digital color photograph beforecomparing the extracted color of each of the plurality of chemical testpads with each respective color trajectory of each chemical test pad. 7.The diagnostic system of claim 4, wherein one of the one or more testresults is protein concentration in the biological sample.
 8. Thediagnostic system of claim 4, wherein one of the one or more testresults is nitrite concentration in the biological sample.
 9. Thediagnostic system of claim 4, wherein one of the one or more testresults is leukocyte concentration in the biological sample.
 10. Thediagnostic system of claim 4, wherein one of the one or more testresults is urobilinogen concentration in the biological sample.
 11. Thediagnostic system of claim 4, wherein one of the one or more testresults is microalbumin concentration in the biological sample.
 12. Thediagnostic system of claim 4, wherein one of the one or more testresults is bilirubin concentration in the biological sample.
 13. Thediagnostic system of claim 4, wherein one of the one or more testresults is glucose concentration in the biological sample.
 14. Thediagnostic system of claim 4, wherein one of the one or more testresults is creatinine concentration in the biological sample.
 15. Thediagnostic system of claim 4, wherein one of the one or more testresults is ketone concentration in the biological sample.
 16. Thediagnostic system of claim 4, wherein one of the one or more testresults is specific gravity of the biological sample.